Skip to content

Prevent S6 startup races: services wait for their config oneshots and depend on base - #689

Merged
jaydrogers merged 4 commits into
serversideup:release/webserver-improvements-and-fixesfrom
LorenzoRogai:fix/root-mode-fpm-nginx-startup-race
Sep 10, 2026
Merged

Prevent S6 startup races: services wait for their config oneshots and depend on base#689
jaydrogers merged 4 commits into
serversideup:release/webserver-improvements-and-fixesfrom
LorenzoRogai:fix/root-mode-fpm-nginx-startup-race

Conversation

@LorenzoRogai

@LorenzoRogai LorenzoRogai commented Jul 22, 2026

Copy link
Copy Markdown

Note

Note from @jaydrogers: I took over this PR to finish it and rewrote this description to match our PR format. The scope grew beyond the original patch (see my comment below for why). @LorenzoRogai's original description is preserved at the bottom of this comment.

Why we created this PR

When a container built from our S6 images runs as root (for example, following the advice in #425), php-fpm sometimes fails on its first start:

ALERT: [pool www] user has not been defined
ERROR: failed to post process the configuration
ERROR: FPM initialization failed

S6 restarts it and the container recovers, so it looks like noise. It's a race. PHP-FPM refuses to start as root without a user in the pool config. Our 5-fpm-pool-user oneshot writes that line, but php-fpm never declared that it depends on the oneshot, so S6 started both at once. On the same boot, nginx and apache2 could also start before 10-init-webserver-config rendered their config.

Rootless containers never hit it because PHP-FPM ignores the user directive when it isn't root. On main, the error reproduces on every root-mode boot.

While fixing the dependency, we found two more things that S6 Overlay documents and we weren't doing:

  1. We used the deprecated flat dependencies file. The s6-overlay README only documents dependencies.d/ directories, and s6-rc marks the flat file deprecated. With the directory format, a dependency is one empty file, so the whole fix is a touch.
  2. None of our services depended on base. The README is direct about it: services that don't depend on base "might have been started earlier, which may cause race conditions - so it's recommended to always make them depend on base."

How to test

serversideup/php-dev:689-*

View the testing images →

To see the race yourself, build this against serversideup/php:8.4-fpm-nginx-alpine (fails) and against a 689- image (passes):

FROM serversideup/php:8.4-fpm-nginx-alpine
USER root
RUN docker-php-serversideup-s6-init
cid=$(docker run -d --rm my-test-image); sleep 6
docker logs $cid 2>&1 | grep -c 'user has not been defined'   # 1 before, 0 after
docker stop $cid

What this PR does

Startup order

  • php-fpm now waits for 5-fpm-pool-user; nginx and apache2 wait for 10-init-webserver-config and php-fpm
  • Every service and every generated oneshot now depends on base, as S6 Overlay recommends

Before (main): php-fpm, nginx, and apache2 start the moment S6 is up, in parallel with the oneshots that write their config.

flowchart LR
    subgraph oneshots [Entrypoint oneshots]
        a[0-container-info] --> b[1-log-output-level]
        b --> c[5-fpm-pool-user]
        b --> d[10-init-webserver-config]
        d --> e[50-laravel-automations]
    end
    subgraph services [Long-running services]
        fpm[php-fpm] --> web[nginx / apache2]
    end
    c -. race .- fpm
    d -. race .- web
Loading

After: every service and oneshot depends on base, and each service waits for the oneshot that configures it.

flowchart LR
    base --> a[0-container-info] --> b[1-log-output-level]
    b --> c[5-fpm-pool-user] --> fpm[php-fpm]
    b --> d[10-init-webserver-config] --> e[50-laravel-automations]
    d --> web[nginx / apache2]
    fpm --> web
Loading

Every node above also lists base directly in its dependencies.d/; those edges are omitted to keep the diagram readable.

S6 Overlay alignment

  • Replaced the flat dependencies files for php-fpm, nginx, and apache2 with dependencies.d/ directories
  • docker-php-serversideup-s6-init now has one add_dependency helper (a mkdir -p and a touch) used for oneshot chaining and for the service dependencies above

Docs

Compatibility

Stock images and rootless containers behave the same. If you appended lines to our old flat dependencies files in your own Dockerfile, S6 now ignores those files once dependencies.d/ exists. Move each line to an empty file in dependencies.d/ instead.


Original description by @LorenzoRogai

Problem

When a container built on the s6 images runs as root, php-fpm and the web server (nginx/apache2) are brought up in parallel with the entrypoint oneshots that configure them — the long-running services declare no dependency on those oneshots.

As root this is a race:

  • php-fpm reads its pool config before 5-fpm-pool-user appends user/group, failing with:
    ALERT: [pool www] user has not been defined
    ERROR: failed to post process the configuration
    ERROR: FPM initialization failed
    
  • the web server starts before 10-init-webserver-config renders its config, e.g. nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (2: No such file or directory).

s6 restarts the crashed longruns, so the container eventually recovers — which is exactly why this is intermittent and hard to reproduce (see #425). But it emits alarming errors on every boot that loses the race, slows startup, and leaves a brief window with no service.

Fix

docker-php-serversideup-s6-init now adds a dependency from each web-facing longrun to the entrypoint oneshot that configures it, after the oneshots are created:

add_startup_dependency php-fpm 5-fpm-pool-user
add_startup_dependency nginx 10-init-webserver-config
add_startup_dependency apache2 10-init-webserver-config

The entrypoint oneshots are chained in alphabetical order, so depending on one transitively waits for all earlier ones (php-fpm5-fpm-pool-user; the web servers → 10-init-webserver-config, which already sits after 5-*). Each dependency is added only when both the service and the oneshot exist, so cli/fpm/frankenphp images — and images where a user removes a script — are unaffected. It does not make services wait on later app oneshots such as 50-laravel-automations, so a failing app hook won't block the web server from starting.

Testing

  • Reproduced as root on 8.4-fpm-nginx-alpine: first boot shows the fpm + nginx errors, then self-heals.
  • With the fix: 3/3 clean boots — no errors, php-fpm/nginx reach "ready to handle connections" on the first attempt.
  • Rootless (default) behavior is unchanged (5-fpm-pool-user is a no-op when not root; the dependency just orders startup).

Refs #425

…-mode startup race

When a container built on the s6 images runs as root, php-fpm and the web
server (nginx/apache2) are brought up in parallel with the entrypoint
oneshots that configure them, because the long-running services have no
dependency on those oneshots.

As root this races:
- php-fpm reads its pool before `5-fpm-pool-user` appends `user`/`group`,
  failing with "ALERT: [pool www] user has not been defined" ->
  "ERROR: FPM initialization failed".
- the web server starts before `10-init-webserver-config` renders its
  config (e.g. nginx: open() "/etc/nginx/nginx.conf" failed).

s6 restarts the crashed services so the container eventually recovers, which
is why the failure is intermittent and hard to reproduce (see discussion
serversideup#425), but it emits alarming errors, slows startup, and leaves a brief window
with no service.

docker-php-serversideup-s6-init now adds a dependency from each web service to
the entrypoint oneshot that configures it, appending to the existing flat
`dependencies` file. The oneshots are chained in alphabetical order, so
depending on one transitively waits for all earlier ones (php-fpm ->
5-fpm-pool-user; nginx/apache2 -> 10-init-webserver-config). Entries are
de-duplicated and appended newline-safely (nginx's shipped `dependencies` has
no trailing newline). Dependencies are only added when both the service and
the oneshot exist, so cli/fpm/frankenphp images and images that remove a
script are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@LorenzoRogai
LorenzoRogai force-pushed the fix/root-mode-fpm-nginx-startup-race branch from 46d9ba2 to c7befe0 Compare July 22, 2026 14:05
Comment thread src/s6/usr/local/bin/docker-php-serversideup-s6-init Outdated
Comment thread src/s6/usr/local/bin/docker-php-serversideup-s6-init Outdated
Comment thread src/s6/usr/local/bin/docker-php-serversideup-s6-init Outdated
Comment thread src/s6/usr/local/bin/docker-php-serversideup-s6-init Outdated
Comment thread src/s6/usr/local/bin/docker-php-serversideup-s6-init Outdated
@jaydrogers
jaydrogers changed the base branch from main to release/webserver-improvements-and-fixes September 10, 2026 15:35
jaydrogers and others added 3 commits September 10, 2026 10:35
…lat dependencies file to dependencies.d, make every user service depend on base as s6-overlay recommends, and make php-fpm, nginx, and apache2 wait for the oneshots that write their config. Fixes root-mode FPM 'user has not been defined' race (serversideup#425, serversideup#689).

Co-authored-by: LorenzoRogai <1665768+LorenzoRogai@users.noreply.github.com>
Co-authored-by: mbrodala <5037116+mbrodala@users.noreply.github.com>
@jaydrogers

Copy link
Copy Markdown
Member

Thanks @LorenzoRogai and @mbrodala! We definitely had a problem here and I want to thank you both for helping me identify and resolve this.

I noticed a few other things so I made a bigger refactor (289b4b0) of this process and gave you both as credit. Here's why:

Why the format changed

While verifying the fix I checked the s6-overlay README and the s6-rc docs. The flat dependencies file we've been using is deprecated. S6 only documents dependencies.d/ directories, where each dependency is an empty file. Once I switched to that, the grep, the newline handling, and the dedupe check all disappeared, because touch is already idempotent. That's the same simplification Matthias was pushing toward, taken one step further. The touch suggestion is in; the printf and long-flag grep suggestions became moot because those lines no longer exist.

Why base was added.

The README is explicit that services which don't depend on base "might have been started earlier, which may cause race conditions." Since this PR is about a startup race, it felt wrong to fix one ordering gap and leave the documented one open.

Verification.

On main, the FPM error reproduced on 3 of 3 root-mode boots. With this branch it's 0 of 3 as root and 0 of 3 rootless, s6-rc-compile accepts the tree, and shellcheck is clean.

One small correction to the original write-up: 10-init-webserver-config does not run after 5-fpm-pool-user. They run in parallel because the init script processes 10-* before 5-* in glob order. The fix still holds because nginx and apache2 depend on php-fpm, which depends on 5-fpm-pool-user.

Thanks again for working together on this!

@jaydrogers
jaydrogers dismissed mbrodala’s stale review September 10, 2026 16:20

I made a bigger refactor and explained why here #689 (comment)

@jaydrogers jaydrogers changed the title fix(s6): make web services wait for their config oneshots (fixes root-mode startup race) Prevent S6 startup races: services wait for their config oneshots and depend on base Sep 10, 2026
@jaydrogers
jaydrogers merged commit 24e60f9 into serversideup:release/webserver-improvements-and-fixes Sep 10, 2026
79 checks passed
@LorenzoRogai

Copy link
Copy Markdown
Author

@jaydrogers thanks for the update! Will this require any changes on our side if we’re using custom s6 scripts?

@LorenzoRogai
LorenzoRogai deleted the fix/root-mode-fpm-nginx-startup-race branch September 11, 2026 15:32
@jaydrogers

Copy link
Copy Markdown
Member

Yes, I ahve that added here: #645

image

Let me know if you have any other questions. Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants